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A consortium of government, industry and academia is currently working to establish 
minimum operational performance standards for Detect and Avoid (DAA) and Control and 
Communications (C2) systems in order to enable broader integration of Unmanned Aircraft 
Systems (UAS) into the National Airspace System (NAS). One subset of these performance 
standards will need to address the DAA display requirements that support an acceptable 
level of pilot performance. From a pilot’s perspective, the DAA task is the maintenance of 
self separation and collision avoidance from other aircraft, utilizing the available 
information and controls within the Ground Control Station (GCS), including the DAA 
display. The pilot-in-the-loop DAA task requires the pilot to carry out three major 
functions: 1) detect a potential threat, 2) determine an appropriate resolution maneuver, and 
3) execute that resolution maneuver via the GCS control and navigation interface(s). The 
purpose of the present study was to examine two main questions with respect to DAA display 
considerations that could impact pilots’ ability to maintain well clear from other aircraft. 
First, what is the effect of a minimum (or basic) information display compared to an 
advanced information display on pilot performance? Second, what is the effect of display 
location on UAS pilot performance? Two levels of information level (basic, advanced) were 
compared across two levels of display location (standalone, integrated), for a total of four 
displays. The authors propose an eight-stage pilot-DAA interaction timeline from which 
several pilot response time metrics can be extracted. These metrics were compared across 
the four display conditions. The results indicate that the advanced displays had faster overall 
response times compared to the basic displays, however, there were no significant 
differences between the standalone and integrated displays. Implications of the findings on 
understanding pilot performance on the DAA task, the development of DAA display 
performance standards, as well as the need for future research are discussed. 
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Acronyms 


ATC 

C2 

CPA 

CSD 

DAA 

FAA 

GCS 

JADEM 

MACS 

NAS 

NMAC 

SC-228 

TSD 

UAS 

VSCS 


Air Traffic Control 

Control and Communications 

Closest Point of Approach 

Cockpit Situation Display 

Detect and Avoid 

Federal Aviation Administration 

Ground Control Station 

Java Architecture for DAA Modeling and Extensibility 
Multi-Aircraft Control Station 
National Airspace System 
Near Mid Air Collision 

RTCA Special Committee 228 Minimum Operational Performance Standards for UAS 
Tactical Situation Display 
Unmanned Aircraft Systems 
Vigilant Spirit Control Station 


I. Introduction 

U NMANNED Aircraft Systems (UAS) have flown in the National Airspace System (NAS) in a limited and 
restricted capacity since the 1990s 1 . Much of the early UAS operations in the NAS were for public purposes 
such as military training and border security, and occurred in remote locations and/or special use airspace. 
However, there has been a recent spike in the demand to operate UAS more widely, for both expanded public 
operations as well as civil and commercial purposes; commercial photography and videography, cargo 
transportation and delivery, communications and broadcasting, disaster response, infrastructure monitoring and 
science are some of the currently proposed applications for UAS. This demand creates the need for new policies to 
support the broader and more integrated operation of UAS in the NAS at an acceptable level of safety. New UAS- 
specific policies will specify the development of new technologies to meet new or existing regulations and 
standards, since current UAS technologies were not developed with compliance to existing airspace operations, 
procedures, regulations, or standards in mind. One major regulatory driver identified by the Federal Aviation 
Administration (FAA) UAS Roadmap 1 , is the development of minimum standards for Detect and Avoid (DAA) and 
Control and Communications (C2) systems, which are currently being addressed by RTCA Special Committee 228 
(SC-228), Minimum Operational Performance Standards for Unmanned Aircraft Systems, a consortium of 
government, industry and academic subject matter experts 2 . 


A. Detect and Avoid (DAA) 

The DAA system is a critical functional area intended to provide the pilot-in-command with the ability to 
self-separate from (i.e., remain well clear of), and avoid collisions with, other aircraft. The DAA functionality is an 
alternate method of compliance to the “see and avoid’’ requirements specified by the Title 14 Code of Federal 
Regulations (14CFR) section 91.113 3 , for a pilot on board a manned aircraft to remain well clear of other aircraft. 
Thus, the DAA system is intended to compensate for the removal of the pilot from the cockpit of an aircraft to a 
control station on the ground. The DAA system notionally consists of both hardware and software components: a 
surveillance sensor (or suite of sensors), data fusion/correlation logic, threat detection and resolution logic, the 
display of traffic information and resolution logic, and control and navigation interfaces to execute a resolution 
maneuver 2 . Since the DAA system introduces both new equipment and new pilot tasks to the entire UAS system, 
new or revised operational rules and standards are needed to ensure that the system and its architectural components 
are designed to an appropriate level of system performance and design assurance in order to meet an airspace safety 
threshold 2 ' 4 . A key assumption for the near term minimum operational performance standards currently being 
established, is that there is a pilot-in-the-loop with the DAA system that has “the ability to command, intervene, 
and/or reject DAA maneuvering under normal operating conditions,” 2 . Under this pilot-in-the-loop assumption, the 
pilot-in-command will directly execute a resolution maneuver via the Ground Control Station (GCS), as opposed to 
having a self separation capability that can execute a resolution maneuver automatically (either with or without pilot 
permission and/or supervision). Thus, one subset of these performance standards will need to address the DAA 
display requirements that support an acceptable level of pilot performance. 
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1. DAA and Pilot Performance 

From a pilot’s perspective, the DAA task is the maintenance of self separation and collision avoidance from 
other aircraft, utilizing the available information and controls within the GCS. The pilot-in-the-loop DAA task 
requires the pilot to carry out three major functions: 1) detect a potential threat, 2) determine an appropriate 
resolution maneuver, and 3) execute that resolution maneuver via the GCS control and navigation interface(s). The 
most critical safety measure of pilot performance on the DAA task is the number of violations of a specified 
separation boundary (or how close the ownship came to another aircraft). The specified separation boundary may be 
a self separation threshold, such as well clear, and/or a collision avoidance boundary, such as a Near Mid Air 
Collision (NMAC). (Typically, the self separation threshold is larger than the collision avoidance threshold and may 
serve to warn the pilot of a potential collision avoidance threshold violation.) 

Another useful quantification of pilot 
performance is measured response. 

Measured response has been used to 
quantify the end-to-end response time for a 
UAS pilot to complete an Air Traffic 
Control (ATC) clearance 5 ' 6 ' 7 . By breaking 
down the end-to-end response into discrete 
stages, from the issuance of the clearance 
until the UAS completes the maneuver, it 
is possible to extract discrete response 
time metrics, such as the time required for 
a pilot to initiate a control input into the 
GCS, or the time required to complete a 
control input in the GCS. A study by Rorie 
and Fern 7 , found these response time 
metrics to be sensitive to differences in 
GCS command and control input 
interfaces. Table 1 presents the six stages 
of pilot-ATC interaction, identified by 
Rorie and Fem, for a pilot responding to 
an ATC clearance. 

Measured response could be adapted for the DAA pilot task by quantifying the end-to-end response time for a 
UAS pilot to complete a self separation or collision avoidance maneuver in response to an alert presented on the 
DAA display. Pilot response time is critical to the maintenance of safe separation from other aircraft - delayed pilot 
responses could result in delayed aircraft maneuvering, which in turn could cause violations of separation 
thresholds. A delayed response may be due to a pilot’s inability to detect a threat, determine an appropriate 
resolution, or execute the resolution. Alternatively, the operational environment may constrain the amount of time 
that the pilot has to detect and/or respond to potential conflicts. Firstly, pilots may be constrained in when they can 
detect threats; late detections may result from encounter geometries (e.g., last minute changes in altitude that result 
in a late “pop-up” threat) and/or limited surveillance ranges for UAS sensors, especially sensors intended to detect 
non-cooperating aircraft in the airspace. Secondly, pilots may be constrained in when they can maneuver against 
threats. For example, ATC may find it unacceptable and disruptive for pilots to maneuver too early against a 
potential threat; thus, considerations for interoperability with ATC must be taken into account. Given potentially 
limited timelines, the DAA display will need to support timely pilot responses to potential threats. 

Response time metrics extracted from a DAA task timeline could be compared across different DAA concepts, 
such as DAA displays. This type of analysis would provide insight into how various DAA concepts or features 
impact pilots’ overall DAA task performance, as well as potential effects on the three separate DAA functions (i.e., 
detect, determine, and execute). In addition, display concepts or features that support faster pilot response could be 
identified. 

2. DAA Displays 

The DAA display plays a critical role in supporting pilot performance on the DAA task. New regulations will 
need to specify the minimum set of information required to achieve a minimum acceptable level of performance to 
meet an airspace safety threshold. The minimum requirements for the DAA pilot display could specify the 
presentation of a minimum set of informational elements such as: ownship and intruder parameters (e.g., heading, 
altitude, airspeed), conflict detection and alerting, and conflict resolution guidance. The minimum information 


Table 1. The six stages of pilot-ATC interaction for a pilot 
responding to an ATC clearance. Reprinted with permission 
from Rorie and Fern 7 . 


Stage 

Description 

Example 

To 

Initial ATC 

“HAWK21, turn left heading 1-2-0, 


Transmission 

vectors for your descent” 

Ti 

Pilot Reply 

“Turn left heading 1-2-0, 
HAWK21” 

T 2 

Pilot Initiates Edit 

Pilot opens editing window 

T 3 a 

Pilot Uploads 1 st 

Pilot incorrectly uploads 110° Hdg 


Edit 

to the aircraft 

T 3b 

Pilot Uploads 

Pilot correctly uploads 120 ° Hdg to 


Final Edit 

the aircraft 

t 4 

UAS Completes 
Maneuver 

HAWK21 reaches 120° Hdg 
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requirements will need to be based on quantitative evidence of acceptable levels of pilot performance, such as those 
discussed above. 

While there have been several recent human-in-the-loop evaluations of traffic displays for UAS 8 ' 9 ' 10 , most have 
focused on military airspace operations rather than civil airspace operations. In addition, only a limited number of 
studies have examined traffic displays in the context of specific information requirements and their effect on pilot 
performance; that is, most studies have examined how to present a pre-determined set of information rather than 
what specific information is required. 

A recent study examined the minimum visual information requirements for a UAS DAA system by comparing 
four display categories with different, and progressively more, information: Position, Direction, Prediction, and 
Rate 11 . The Position display provided the pilot participants with instantaneous intruder positions and included 
aircraft identification, range, bearing, relative altitude, absolute altitude and numeric range. The Direction display 
added directionality as well as a heading chevron, numeric heading, and a vertical trend arrow. The Prediction 
display added yellow and red alert color-coding, and 30 second dead-reckoning vector lines to intruders. Finally, the 
Rate display added ground speed, history trails, and climb/descent rates. This study found that the Prediction display 
performed as good as, or better, than the Position and Direction displays, and no different than the Rate displays 
across a number of performance and workload metrics. Perhaps most importantly, the Prediction display resulted in 
significantly less NMACs than the two lower level displays (Position and Direction), and was not significantly 
different from the Rate display. The authors concluded that the information included in the Prediction display was 
the minimum visual information required for a DAA pilot display. 

A survey conducted by Draper, Pack, Darrah, Moulton, & Calhoun 12 , found that the majority of pilots surveyed 
indicated that the following information should be present at all times on a DAA display: intruder identification, 
intruder location, intruder relative position, intruder threat/alert level, DAA task priorities and status, DAA 
maneuver recommendations, flight restrictions, weather, navigation data, and visual alerts. 

While Friedman-Berg et al. 11 and Draper et al. 12 sought to establish a minimum information set, Bell 13 compared 
a basic display (similar to the Direction display described in the Friedman-Berg et al. study) to two “advanced” 
concepts: a display that depicted the relative Closest Point of Approach (CPA) between ownship and intruder, and a 
display that depicted ownship avoidance areas with polygon shapes. This study found no significant differences in 
the frequency of violations of the defined well clear threshold between the three display concepts, however, there 
was a significant difference in the duration of violation events - the basic display had significantly longer violation 
durations than both of the other two advanced displays. 

Together, these three studies provide a starting point for the systematic evaluation of candidate DAA displays, 
specifically with respect to determining the minimum information requirements for such displays. In addition to 
specifying the required informational elements, DAA display standards could address the location of the pilot 
display within the GCS, for example, whether the DAA display can be a standalone, separate display, or whether it 
needs to be integrated into the moving map, or tactical situation display. Display location has the potential to 
significantly impact pilot performance in maintaining self separation and collision avoidance from other aircraft. A 
standalone display is considered a near term technology solution for existing UAS, because it is easier to develop, 
certify and field an independent, separate display compared to modify existing GCS software. However, the 
standalone display concept has some disadvantages. First, a standalone traffic display is unlikely to be integrated 
with the GCS command and control interface, thus a pilot would have to identify a potential threat and resolution 
maneuver, and then translate that to the command and control interface on another display in order to execute the 
maneuver, and confusion could arise if the displays are at different zoom levels or orientations (e.g., track up versus 
north up). In addition, a standalone display is likely to be produced by a different manufacturer than the GCS, which 
could result in inconsistencies in the presentation of similar information between the DAA and GCS displays. 
Finally, pilot response times could be slower as pilots have to switch attention (and possibly interaction) between 
two different displays (and control inputs). While a DAA display that was integrated into the GCS moving map 
could overcome several of these disadvantages, in addition to the large increase in overhead in terms of resources to 
develop, a major risk of an integrated display is increased clutter on the moving map display. Thus, display location 
in addition to minimum display information, are critical issues that need to be understood within the context of pilot 
performance on the DAA task. 

The simulation experiment detailed in this paper is the first of several planned experiments intended to quantify 
the minimum operational performance standards for DAA displays to support safe operation of UAS in the NAS. 
The goal of the planned experiments is to systematically evaluate the effect of candidate DAA display concepts 
(e.g., information elements, maneuver guidance, location, etc.) on pilots’ response time and ability to maintain safe 
separation from other aircraft. The purpose of the present study was to examine two main questions with respect to 
DAA pilot display considerations for self separation and collision avoidance. First, what is the effect of a basic or 
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minimum information display compared to an advanced information display on pilot performance? Second, what is 
the effect of display location on UAS pilot performance? This paper focuses specifically on pilots’ measured 
response, adapted from the pilot- ATC interaction timeline shown in Table 1, while responding to self separation and 
collision avoidance encounters. Santiago and Mueller 4 report on well clear violation metrics for this experiment. 

The authors proposed three main hypotheses with respect to these questions: 

1) Displays with an advanced information set will have faster response times overall than those with a minimum 

information set given the additional information provided by advanced displays to help the pilot detect 
potential threats and determine appropriate resolutions. 

2) Integrated displays will support faster response times than standalone displays given the extra time required 

by a standalone display to translate traffic information from one display into a maneuver response on 
another display. 

3) Given the above hypotheses, an advanced, integrated display will have faster response times than all other 

displays. 


II. Method 


A. Participants 

Twelve active duty RQ-4 pilots (M = 39 years of age) were recruited for this experiment. Participants had an 
average of 216 hours of experience flying UAS in combat and non-combat military operations. Eight of the 
participants had prior experience flying UAS in civil airspace, each with an average of 60 hours. A single retired 
ATC served as a confederate for the study. 

B. Simulation Environment 

1. Ground Control Station 

Participants were situated at a UAS GCS 
containing two pieces of software distributed 
across four separate monitors, the Vigilant 
Spirit Control Station (VSCS) and the Cockpit 
Situation Display (CSD). Figure 1 shows the 
UAS GCS display set up. VSCS, developed by 
the Air Force Research Faboratory, is a mature 
GCS operator interface designed to support the 
control of UAS and their associated payloads 14 . 

In this study, VSCS generated three separate 
pilot displays in the ground station: a Tactical 
Situation Display (TSD), a health and status 
panel, and a simulated out-the-window nose 
camera display. The TSD (shown in Fig. 2) 
served as the pilot’s primary display, providing 
ownship and route information, a moving map, 
and navigation and control interfaces. The TSD 
supported two separate vehicle control 

interfaces. The first, a waypoint-editing 

interface, allowed pilots to modify the assigned 
altitude or location of any waypoint on their 
mission route. The second, an autopilot editing interface, allowed pilots to enter into altitude, speed, and heading 
holds without modifying their mission route. Heading holds could be executed through numerical inputs to a 
steering command window or through interaction with a graphical compass rose interface (shown in Fig. 3). The 
compass rose interface allowed pilots to drag a heading bug to their desired direction rather than enter the value 
manually. Altitude and speed holds, however, could only be executed using numerical inputs to the steering 
command window. Pilots uploaded changes to the aircraft by pressing a “Send” button within the steering command 
window. As will be discussed in detail below, the TSD was also configured to display surrounding traffic, alerting 
information, and conflict resolution tools in select conditions. Unless otherwise stated, the TSD was restricted to the 
functionality described here. 

A health and status panel was positioned to the right of the TSD and contained subsystem information, telemetry 
data, a chat client, and an electronic checklist. The third VSCS display was a simulated out-the-window nose-camera 
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Figure 1. UAS GCS. CSD (not displayed), bottom left; TSD, 
bottom center; out-the-window view, top center; health and 
status panel, bottom right. 






Figure 2. Vigilant Spirit Control Station tactical situation display Figure 3. Vigilant Spirit Control 
(AFRL/RH). Distribution A: Approved for public release; Station’s compass rose interface, 

distribution unlimited, 3/18/2013; 88ABW-2013-1303. 

view positioned directly above the TSD. This monitor 
included synthetic terrain and an integrated head up 
display with current airspeed, altitude and heading 
information. All interaction with the three VSCS 
components occurred through standard mouse and 
keyboard inputs. 

A fourth monitor was populated by the CSD (shown in 
Fig. 4), a 3D volumetric cockpit display of traffic 
information developed by the Flight Deck Display 
Research Laboratory at NASA Ames Research Center 1 '. 

The CSD, positioned directly to the left of the TSD on a 
separate monitor, was configured to display ownship 
information, surrounding traffic and alerting information, 
and, in select conditions, conflict detection tools. Pilot 
interaction with the CSD, while limited, was enabled 
through standard keyboard and mouse inputs. Pilots were 
able to adjust the display range of the CSD by using the 
mouse’s scroll wheel within the boundary of the 
volumetric display or by using a range dial on an external 
menu. The CSD had a minimum display range of lOnm 
and a maximum display range of 640nm. Along with Figure 4. The 
intruder and ownship information, the CSD provided pilots developed by 
with range rings and heading markers along the edge of the Laboratory, 
display. For the purposes of this experiment, the CSD was 
restricted to a 2D top-down view and an ego-centric, north- 
up orientation. 

2. Detect and Avoid System 

The DAA system utilized in this experiment was provided by the Java Architecture for DAA Modeling and 
Extensibility 16 . JADEM served as this study’s surveillance and threat detection system. The software was configured 
to replicate ADS-B, with a lateral sensor range of 80nm and a vertical sensor range of +/- 5000 feet. Aircraft outside 
of this range were not displayed to the pilot. JADEM also contained conflict detection logic that evaluated and 
prioritized surrounding traffic according to their predicted threat level with ownship. To calculate the predicted 
threat level, JADEM compared ownship’s known intent to surrounding traffic, extrapolating the intruder’s future 
position assuming their constant velocity. The spatial and temporal thresholds used to determine threats, along with 
their associated threat alerting levels, are described in further detail below. Finally, JADEM also provided pilots in 
the advanced display conditions with text-based directive guidance through its conflict resolution algorithm, 
Autoresolver-AD 16 . Autoresolver-AD presented pilots with a recommended maneuver, in the form of a text box that 



Cockpit Situation Display (CSD), 
the Flight Deck Display Research 
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was calculated as inducing the least amount of delay for resolving the active threat. If Autoresolver-AD was unable 
to generate a threat-free maneuver, the algorithm presented pilots with a maneuver that maximized the horizontal 
miss distance between ownship and the intruder. If at any point Autoresolver-AD computed a more effective (i.e., 
safer or more efficient) maneuver, pilots were given the ability to replace the previous recommendation with the 
latest solution. Autoresolver-AD was disabled in the basic display conditions. 

3. Traffic Simulation 

The Multi-Aircraft Control Station (MACS) provided the air traffic simulation environment for this study 17 . 
MACS was used to generate simulated traffic targets, the confederate controller’s display, and two pseudo pilot 
stations. An en route ATC display provided the confederate controller with the ability to realistically manage all 
traffic within the designated experimental sector (Oakland Center ZOA 40/41). Two pseudo pilot stations enabled 
confederate pilots to take control of, and respond as, any manned aircraft in the simulated airspace. All experimental 
participants communicated over a single voice IP communication application. 

C. Experimental Design 

This study utilized a within-subjects, repeated measures factorial design to compare the effect of information level 
and display location on UAS pilots’ performance on maintaining self separation and collision avoidance from other 
aircraft while operating in civil airspace. Two levels of information (basic, advanced) were compared across two 
levels of display location (standalone, integrated), thus a total of four displays were examined: 1) basic standalone, 
2) basic integrated, 3) advanced standalone, and 4) advanced integrated. 

1. Information Level 

The results from Friedman-Berg et al. 11 and Draper et al. 12 discussed above, were crosschecked against existing 
relevant references and documents (e.g., DO-317 Minimum Operational Performance Standards for Aircraft 
Surveillance Applications System 18 ) in order to come up with a set of minimum information requirements that could 
be evaluated in this experiment as the basic condition. The information set for the basic display condition is listed in 
Table 2. 


Table 2. The minimum information set for the basic display condition. 


Intruder Information 

Visibility 

Ownship Information 

Visibility 

Location 

Always visible 

Location 

Always visible 

Range 

Always visible 



Bearing 

Always visible 

Trajectory 

Always visible 

Heading 

Always visible 

Heading 

Always visible 

Relative Altitude 

Always visible 

Altitude 

Always visible 

Vertical Trend 

Always visible 

Vertical Trend 

Always visible 

Heading Predictor 

Always visible 



Alert/Threat Level 

Always visible 



Vertical Velocity 

Within data tag 

Vertical Velocity 

Within data tag 

Absolute Altitude 

Within data tag 



Ground Speed 

Within data tag 

Ground Speed 

Within data tag 

Aircraft ID 

Within data tag 




The visibility of each information element was also specified, that is, whether it should be always visible on the 
DAA display, or whether it could be shown in a data tag. Information shown within a data tag was only displayed if: 
1) the data tag was selected for an intruder with a Proximal or Preventative threat level (see below), or 2) an aircraft 
had a self separation or collision avoidance threat level (at which point the data tag information was automatically 
visible). 

Table 3 presents the multi-level alerting structure that was used in the basic display condition. Threat level was 
based on the location of and time to the closest point of approach (CPA) between the ownship and an intruder 
aircraft. Both the location and time thresholds had to be met for an intruder to be assigned a threat level. Location of 
the CPA was measured by both the lateral distance, in nautical miles (nm), and vertical distance, in feet (ft), to the 
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predicted CPA location. The time to CPA was measured in seconds (sec). For this experiment, the collision 
avoidance and well clear thresholds were treated as equivalent. 


Table 3. The multi-level alerting structure used in the basic display condition. 


Alert/Threat Level 

CPA Distance from Ownship 

Time to CPA 

Color 

Proximal 

Lateral 

> 2 nm 

Vertical 

> 900 ft 

N/A 

Grey 

Preventative 

< 2 nm 

< 900 ft 

< 120 sec 

White 

Self Separation 

< 1 .2 nm 

< 900 ft 

< 1 10 sec 

Yellow 

Collision Avoidance 

< 0.8 nm 

< 400 ft 

< 40 sec 

Red 


The advanced display condition contained all of the information elements within the basic display condition, but 
added additional features to assist the pilot in maintaining self separation and collision avoidance from other aircraft. 
These additional features included: an additional predicted collision avoidance alerting level, a depiction of intruder 
and ownship predicted CPA location, a 0.8 nm collision avoidance or well clear threshold “ring,” a vertical situation 
display, a single maneuver recommendation, and trial/vector planning tools. These tools are discussed further under 
each specific display description below. 

2. Display Location 

For this experiment, the standalone DAA display condition could only receive ownship state and trajectory 
information from the navigation system, and could not send any information to it (e.g., to the command and control 
interface). The integrated display condition saw the DAA display features integrated directly into the moving map 
display of the GCS. The DAA display was also integrated with the navigation system. Where possible, display 
features were integrated with the command and control interface. The four separate DAA displays are described 
below. 

3. Basic Standalone 

The basic standalone DAA display condition 
presented pilots with the minimum information set 
(see Table 2) within the CSD. In order to replicate a 
completely independent system, the standalone 
display was configured to receive information from 
the UAS navigation system but was not capable of 
sending any information directly to the aircraft. All 
command and control changes to the aircraft were 
required to be made through the TSD. As shown 
above in Fig. 4, the basic standalone display 
provided pilots with minimal ownship, intruder, and 
alerting information. No guidance or conflict 
resolution tools were present in this display 
condition. 

4. Basic Integrated 

The basic integrated DAA display condition 
presented pilots with the same minimum 
information set but within the VSCS TSD (Fig. 5). 

This configuration allowed pilots to view traffic 
within their primary display, collocated with their 
vehicle control interfaces. The integrated display 
also allowed pilots to open and/or reposition data 
tags anywhere within the map area. No guidance or 
conflict resolution tools were present in this display 
condition. 



♦02 4 ’ i 
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Figure 5. The basic integrated DAA display with two active 
self separation alerts. 
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5. Advanced Standalone 

In the advanced standalone DAA display 

condition (Fig. 6), pilots were provided advanced 
alerting information and resolution tools within 
the standalone display to aid in assessing and 
resolving conflicts. These advanced features were 
provided in addition to the minimum information 
set employed by the basic display conditions. 

Pilots were not required to use or follow the 
guidance provided by the resolution tools included 
in this condition. 

6. Advanced Alerting Information. 

CPA Location . At the onset of a self separation 

alert, the predicted physical location of the 
intruder’s CPA to ownship was depicted by a 
color-coded circle. The CPA location remained on 
the display as long as a threat was active. 

Time to CPA . A countdown timer was 
triggered by the onset of a self separation alert, 
indicating the time remaining until CPA was 
reached. The time was displayed in the bottom 
left-hand corner of the display and disappeared as 
soon as an alert was cleared. 

Predicted Severity . An additional predicted 
collision avoidance alert level was introduced to 
show severity (shown in Table 4). The predicted 
collision avoidance alert level was intended to aid 
pilots in discriminating between critical and non- 
critical self separation threats. The existence of 
larger spatial thresholds for self separation alerts 
compared to collision avoidance alerts meant that 
some self separation alerts never progressed to 
collision avoidance alerts. The predicted collision avoidance alert level indicated critical self separation alerts (that 
were predicted to progress to a collision avoidance alert) by coloring the border of the CPA circle red. For less 
critical self separation alerts (i.e., those not predicted to become collision avoidance alerts), the CPA circle remained 
solid yellow. 

Table 4. The multi-level alerting structure used in the advanced display conditions. 



Figure 6. Screenshot of the advanced standalone DAA 
display condition during a self separation alert. The 
maneuver recommended by Autoresolver-AD is shown in 
the upper right-hand box (“Descend to: 14500 Feet”), the 
lateral trial planner is indicated by the magenta colored 
flight plan, the vertical trial planner (i.e., altitude tape) is 
located on the lower right side of the display, and the time 
to the predicted closest point of approach with the self 
separation alert is located in the bottom left. 


Alert/Threat Level 

CPA Distance from Ownship 
Lateral Vertical 

Time to CPA 

Color 

Proximal 

> 2 NM 

> 900 FT 

N/A 

Grey 

Preventative 

< 2 NM 

< 900 FT 

< 120 SEC 

White 

Self Separation 

< 1.2 NM 

< 900 FT 

<110 SEC 

Yellow 

Predicted Collision Avoidance 

< 0.8 NM 

< 400 FT 

<110 SEC 

Yellow, Red Border 

Collision Avoidance 

< 0.8 NM 

< 400 FT 

< 40 SEC 

Red 


7. Advanced Resolution Tools 

Recommended Maneuvers . At the onset of a self separation alert, Autoresolver-AD provided pilots with a 
recommended resolution maneuver. The maneuver appeared in a text box in the upper right hand of the display. If 
Autoresolver-AD computed a more effective maneuver, a “Refresh” button flashed continuously at the bottom of the 
display. When pressed, the new resolution maneuver replaced the previous one. 

Trial Planning Tools . Two separate trial planning tools, each intended to give pilots the ability to ‘test’ various 
heading and altitude vectors by providing immediate feedback as to the quality of a proposed maneuver, were 
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included. The tools were engaged automatically during self separation alerts and populated with the maneuver 
recommended by Autoresolver- AD. However, pilots were also able to engage the trial planning tools manually by 
selecting a dedicated button (“RAT”) at the bottom of the display. 

The lateral trial planning tool allowed pilots to superimpose a ‘proposed’ route line on top of their active route. 
The proposed route line could be manipulated without any impact to their active route. Using the superimposed 
route line, pilots could trial plan different heading vectors by clicking and dragging on a waypoint off the nose of the 
aircraft. As pilots moved a proposed waypoint away from their active route, a heading readout appeared adjacent to 
the waypoint, informing pilots of the exact heading vector being trial planned. The proposed route line was color- 
coded based on the predicted alert level for the trial-planned heading. Proposed headings that were predicted to lead 
to only proximal alerts turned the line magenta, while headings predicted to lead to at least one preventative alert 
turned the line white. Magenta was used instead of the corresponding grey color for proximal alerts, in order to be 
more visually salient to the pilot and distinctive from the actual current trajectory. Proposed headings that were 
predicted to lead to at least one self separation alert or at least one collision avoidance alert turned the route line 
yellow or red, respectively. While trial planning, halos appeared around intruders that were predicted to become a 
proximal, self separation or collision avoidance alerts. The halos were color-coded according to the predicted alert 
level of the associated intruder. 

Similar to the lateral tool, the vertical trial planning tool allowed pilots to probe different altitudes for their 
relative safety. In order to trial plan different altitudes, pilots manipulated an altitude tape positioned in the bottom 
right-hand corner of the display. The altitude tape included three different altitude ‘bugs’: a current altitude bug, a 
commanded altitude bug, and a trial plan altitude bug. The altitude tape was centered on the trial plan altitude bug, 
allowing pilots to test various altitudes by clicking and dragging the altitude tape’s surface. The trial plan altitude 
bug and the border of the altitude tape were then color coded using the same alerting logic described for the lateral 
trial planning tool. 

8. Advanced Integrated 

In the advanced integrated DAA display condition, pilots were provided advanced alerting information and 
resolution tools within the VSCS TSD to aid in assessing and resolving conflicts (Fig. 7). These advanced features 
were provided in addition to the basic information and alerting set employed by the basic display conditions. As 
with the advanced standalone display condition, pilots were not required to use or comply with the alerting and 
guidance information. 



Figure 7. Screenshot of the advanced integrated DAA display condition with an active predicted collision 
avoidance threat. The maneuver recommended by Autoresolver-AD is shown in the upper box (“Fly Heading 
122”), the lateral trial planner is indicated by the arrow pointing to heading 122 off the nose of the ownship 
icon (center), the vertical trial planner is located on the far right side of the TSD, and the vertical situation 
display is shown in the lower quarter of the display. 
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9. Alerting Information 

CPA Location . The predicted physical location of a self separation threat’s CPA to ownship was presented using 
a color-coded circle on the display. This condition also allowed pilots to display ownship’s CPA to the intruder 
aircraft by hovering over the threat’s aircraft icon. The CPA locations automatically disappeared once an alert was 
cleared. 

Time to CPA . A countdown timer to CPA was triggered by the onset of a self separation alert. The timer was 
displayed in the data tag of the relevant aircraft. This data block field was empty when the target was not an active 
threat to ownship. 

Predicted Severity . The additional predictive collision avoidance alerting level was employed in this display 
condition as well (see Table 4). Whereas a red border on the CPA circle in the advanced standalone condition 
indicated the predicted severity, critical self separation alerts in the advanced integrated condition were indicated 
using a red border around both the CPA circle and the threat’s aircraft icon. 

Well Clear Ring : The presence of a self separation alert enabled the appearance of a “well clear ring” around 
ownship. With a radius of 0.8nm, the ring gave pilots a visual reference to the lateral collision avoidance threshold. 

Vertical Situation Display . A panel at the bottom of the TSD displayed a vertical profile of traffic +/- 1,000 feet 
vertically from ownship. The Vertical Situation Display (VSD) showed traffic icons, heading predictors, CPA 
location, and appropriate color-coding for alert level. Range rings within the VSD were designed to align with the 
range rings on the TSD. 

10. Advanced Resolution Tools 

Recommended Maneuvers . At the onset of a self separation alert, Autoresolver-AD provided pilots with a 
recommended maneuver. The text-based recommendation was displayed in the upper-right hand corner of the TSD. 
If Autoresolver-AD computed a more effective maneuver, a “Refresh” button appeared directly below the 
recommended maneuver text. When pressed, the new resolution maneuver replaced the previous one. 

Trial Planning Tools . As with the advanced standalone condition, two separate trial planning tools, lateral and 
vertical, were included in the display. While the tools’ overall function was still to allow pilots to test various 
heading and altitude vectors, their implementation had to be modified to allow for integration with the vehicle 
controls. Once again, the trial planning tools were automatically engaged at the onset of a self separation alert and 
populated with the maneuver recommended by Autoresolver-AD. The tools could also be launched manually in the 
absence of an active self separation alert. 

The lateral trial planning tool was integrated into the TSD’s autopilot interface. At the onset of a self separation 
alert the compass rose automatically opened on the TSD and a vector arrow appeared, extending from ownship to 
the edge of the compass rose in the direction of the recommended Autoresolver-AD maneuver. As pilots dragged the 
heading bug, the collocated arrow gave instantaneous feedback as to the quality of the proposed heading. As with 
the proposed route line in the advanced standalone condition, the vector arrow was color-coded based on the safety 
level of the heading being probed. Proposed headings that were predicted to lead to only proximal alerts turned the 
vector arrow green, while headings predicted to lead to at least one preventative alert turned the arrow white. 
Proposed headings that were predicted to lead to at least one self separation alert or at least one collision avoidance 
alert turned the vector arrow yellow or red, respectively. Since the vector arrow was integrated into the autopilot 
interface, pilots could directly send any proposed heading holds up to the aircraft by pressing “Send” in the 
autopilot’s steering command window. 

The vertical trial planning tool utilized an altitude table permanently displayed on the far right side of the TSD. 
The altitude table consisted of five discrete altitude options: one at the current altitude, two 1,000 feet above the 
current altitude (in 500 foot increments), and two 1,000 feet below the current altitude (also in 500 foot increments). 
The same color-coding scheme as described for the lateral trial planning tool was used for the altitude table to 
indicate the predicted safety level of each altitude option. Each altitude option was selectable and tied to the TSD’s 
autopilot interface. When pressed, the selected altitude was pushed to the steering command window. Pilots could 
then upload the new altitude to the aircraft by pressing “Send” within the steering command window. While trial 
planning with the lateral or vertical tools, halos appeared around intruders that were predicted to become a proximal, 
self separation or collision avoidance alerts. The halos were color-coded according to predicted alert level. 

11. Pilot Task 

Pilots were tasked with operating a simulated MQ-9 Reaper, “HAWK21,” along one of two pre-filed flight paths 
within Oakland Center airspace (ZOA 40/41). Pilots flew under instrument flight rules (IFR) and were responsible 
for navigating the aircraft and responding to a variety of scripted health and status tasks. These secondary tasks 
included responding to requests for status information (e.g., current fuel level) in a chat client and completing 
electronic checklists in response to aircraft system malfunctions. Pilots were also instructed to monitor their traffic 
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display for potential safety of flight concerns. If a safety of flight threat was discovered, they were instructed to 
notify air traffic control of their desire to maneuver (time permitting) and to upload the change to the aircraft. They 
were encouraged to minimize their deviation from the flight plan and to coordinate a return to their mission route 
and/or altitude with air traffic control as soon as practical. 

12. Scenarios 

Pilots flew two different mission routes, a “Fire Line” mission and a “Coastal Watch” mission. Each route 
started with HAWK21 already at mission altitude, flying towards its second programmed waypoint. There were no 
scripted altitude changes for either of the mission routes. The Fire Line mission route was level at 12,000 feet and 
Coastal Watch mission route remained level at 14,000 feet. Two different manned traffic scenarios were scripted to 
run alongside the two mission routes. Both traffic scenarios were developed by an ATC subject matter expert and 
designed to provide equivalent pilot workload. In each scenario, eight intruders were scripted to progress to a self 
separation then collision avoidance alert, absent of pilot action, while four different intruders were scripted to 
progress only to a preventive alert. All encounters were built for a single intruder, however, dynamic changes to the 
surrounding traffic made it possible for multiple intruders to occur simultaneously. Live traffic data was referenced 
in order to help simulate traffic patterns and densities that were representative of a busy day at Oakland Center. 

D. Procedure 

1. Training 

Participants first completed an informed consent form and a demographics form, which elicited information 
regarding their experience in manned and unmanned aviation. This was followed by a brief overview of the day’s 
schedule and an introduction to the pilot tasks. Pilots then underwent extensive training on the basic functionality of 
VSCS. This included practice on how to use the TSD’s vehicle control interfaces as well as how to perform the 
various health and status tasks that would be present during the experimental trials. Pilots concluded this portion of 
the training with a 20-minute practice scenario. Pilots received hands-on training and completed additional 20- 
minute practice scenarios prior to all subsequent display conditions. 

2. Experimental Trials 

Participants completed four, 37-minute experimental trials. All participants received the four different DAA 
display conditions described above: basic standalone, basic integrated, advanced standalone, and advanced 
integrated. The presentation of the display conditions was counterbalanced across participants to account for order 
and learning effects. Following each experimental trial, participants completed the NASA Task Load Index (TLX) 19 
and a post-trial subjective questionnaire, which focused on the unique display elements of the preceding condition. 
A post-simulation questionnaire and debrief followed the final experimental trial. 

Ill, MEASURES 

Pilots’ measured response for the DAA task was modified from the pilot-ATC interaction measured response 
timelines utilized in previous studies 5,6,7 . The pilot-DAA interaction measured response timeline was extracted from 
time stamps at eight discrete and operationally relevant stages of a pilot’s response to a self separation (yellow and 
yellow with red border) or collision avoidance threat alert (red). A response encounter was defined as an interaction 
with a single intruder, starting from the appearance of a self separation or collision avoidance alert until these alert 
levels were removed and the intruder returned to a preventative or proximal alert level (see Table 3 for the multi- 
level alerting stages). A single alert encounter could have multiple self separation and collision avoidance alerts 
associated with it, however, it was only considered a “new” encounter if the self separation or collision avoidance 
alert was removed and then appeared again (i.e., due to changing trajectories of one or both aircraft). If a pilot 
completed any of the stages in the pilot-DAA timeline after the appearance of a self separation or collision 
avoidance alert, the measures for that response was logged. 

Table 5 describes each of the eight stages of the pilot-DAA interaction timeline. (For the purposes of this study, 
inputs to the command and control interface in order to input a resolution maneuver are termed “edits.”) The time 
stamps for each stage were collected from a variety of data sources including raw MACS, VSCS, and JADEM 
output files, voice recordings and logs, and screen recordings of the pilot display. Where possible, data sources were 
crosschecked with each other, and screen recordings were referenced to provide context of the results. 
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Table 5. Stages of the pilot-DAA interaction timeline. 


Stage Description 

T 0 DAA (self separation or collision avoidance) alert appears on the display 

T i Pilot notifies ATC and requests a maneuver clearance 

T 2 ATC provides maneuver clearance 

T 3 Pilot initiates an edit in GCS to maneuver 

T 4a Pilot uploads 1 st maneuver to aircraft 

T 4b Pilot uploads final maneuver to aircraft 

T 5 Traffic alert is removed from display 

T 6 UAS completes maneuver 


A number of pilot response time metrics were calculated based on the time-stamped stages collected in Table 5. 
The metrics reported here focus on pilots’ interaction with the DAA system, including the traffic displays (detection 
of threats and determination of a resolution) and the control and navigation interface (execution of a resolution 
maneuver); these metrics are most critical to understanding the effect of display configurations on pilot 
performance. All response times were calculated in seconds (s). Figure 8 depicts the temporal relationship between 
reported metrics. 
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Figure 8. Temporal relationship of the pilot measured response stages and response time metrics. 


Notification Time (T j - Tg). A measure of the time it takes a pilot to notify ATC of the appearance of a self 
separation or collision avoidance alert and the need to execute a maneuver. Calculated as the difference between the 
first appearance of the alert and the beginning of the pilot’s transmission on the radio to notify the controller. 

Initial Response Time (T 3 - Tg). A measure of the time it takes a pilot to initiate a maneuver response, or edit, in 
the GCS command and control interface in response to a traffic display alert. Calculated as the difference between 
the first appearance of the alert and the start of an edit. 

Initial Edit Time (T 4a - T 3 ). A measure of the time it takes a pilot to input an initial edit into the GCS in order to 
maneuver in response to a traffic alert. Calculated as the time between initiating an edit and the first upload to the 
aircraft. This metric is only relevant if a pilot uploaded multiple edits to the aircraft; when pilots made only one edit, 
initial edit time is equivalent to total edit time. 

Total Edit Time (T 4 b - T 3 ). A measure of the time it takes a pilot to complete an edit into the GCS in order to 
maneuver in response to a traffic alert. Calculated as the time between initiating an edit and the final upload to the 
aircraft. The final upload is assumed to be the pilot’s final resolution decision. 

Total Response Time (T 4h - Tg). A measure of the time it takes a pilot to upload a final maneuver resolution to the 
aircraft in response to a traffic alert. Calculated as the time between the initial appearance of the traffic alert and the 
final maneuver upload to the aircraft. 


IV. RESULTS 

Each of the five metrics listed above were analyzed utilizing a 2 (information level: basic, advanced) X 2 (display 
location: standalone, integrated) repeated measures Analysis of Variance (ANOVA) with Bonferroni pairwise 
corrections for the main effect post hoc comparisons. An alpha level of .05 was used for all analyses. Significant 
interactions were analyzed using T test comparisons. Pilots responded to a total of 261 discrete alerts, 251 alerts 
appeared initially at the self separation threat level, the remaining ten alerts appeared initially at the collision 
avoidance threat level. 
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A. Notification Time 

The main effect of information level on pilot notification time approached statistical significance, F( 1, 11) = 
4.432 ,p = .059. The advanced displays (M = 25.99; SE = 2.78) appeared to support faster pilot notification times 
than the basic displays (M= 32.16; SE = 4.85). There was no significant main effect of display location, F(l,ll) = 
.140, p > .05, nor was there a significant interaction, Z^(l, 11) = .018,/> > .05. 

B. Initial Response Time 

There was not a main effect of information level on initial response time, F( 1, 11) = .459, p > .05. However, the 
main effect of display location on initial response time approached significance, F{ 1, 11) = 4.635, = .054. Pilots’ 

initial response time appeared faster for the standalone displays (M = 16.78; SE = 1.80) compared to the integrated 
displays (M = 21.85; SE = 2.20). The interaction between information level and display location was not 
significant, ^(1, 11) = .239 , p > .05. 

C. Initial Edit Time 

There was a significant interaction of information level and display location on pilots’ initial edit time, F(l, 11) 
= 13.851, p < .01. Post hoc pairwise T tests revealed a significant difference in initial edit time between the basic 
(M= 17.11; SE = 3.11) and advanced (M = 5.51; SE = 3.42) displays for the integrated display condition, f(ll) = 
3.449, p < .01. However, the difference between basic and advanced was not significant for the standalone 
condition displays, t(ll) = 1.126, p > .05. The difference between basic and advanced for the integrated displays 
was large enough to result in a significant main effect of information level on initial edit time, F( 1, 11) = 8.972, p < 
.05. There was not a significant main effect of display location on initial edit time, F(l, 11) = .139 ,p > .05. 

D. Total Edit Time 

Information level had a significant effect on total edit time, 

F(l, 11) = 1 1.821, < .01 (Fig. 9). The advanced displays (M 

= 13.18; SE = 2,48) had significantly shorter edit times than 
the basic displays (M = 22.12; SE = 3.14). There was not a 
significant main effect of display location on total edit time, 

F(l, 11) = 1.192, p > .05, nor was there a significant 
interaction between information level and display location, 

F(l, 11) = 2.804,/? >.05. 


E. Total Response Time 

There was a significant main effect of information level on 
total response time, F( 1, 11) = 6.619 ,p < .05 (Fig. 10). On 

average, the advanced display condition (M = 33.98; SE = Figure 9. Total edit time by information 

3.34) was 13.79s faster than the basic display condition (M = level. 

41.77; SE = 3.53). The main effect for display location and the 
interaction were both non-significant, F( 1, 11) = .633 and 
F(l, 11) = 2.472, respectively,/? > .05. 


V. DISCUSSION 

The primary purpose of a DAA display for UAS is to 
enable pilots to carry out three primary DAA functions: 
detect potential threats, determine appropriate resolution 
maneuvers, and execute those maneuvers via the GCS control 
and navigation interface. A number of discrete stages mark 
key pilot activities along the pilot-DAA timeline (Table 5). 

The eight stages map roughly, though not perfectly, to the 
three DAA functions a pilot is responsible for. The execute 
function has the most clear mapping to the final upload (T 4b ). 

The detection function can be measured as some time after 
the first alert, T 0 , and before the next stage that the pilot 
executes, for example, notifying ATC of the need to maneuver (Ti) or initiating a maneuver response in the GCS 
(T 3 ; pilots did not always request a clearance prior to maneuvering). The determine function is the most difficult to 
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Figure 10. Total response time by information 
level. 
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map, pilots can be actively determining an appropriate maneuver any time between when the alert appears and the 
final maneuver is executed, essentially the entire timeline that the pilot is engaged in the DAA task. Despite the 
challenge of directly mapping the eight timeline stages to the pilot’s DAA functions, a number of pilot performance 
metrics can be extracted and provide insight to pilots’ activities during these stages. The above results show that 
these metrics are sensitive to various display configurations, especially differences in information across 
configurations. 

As hypothesized, advanced information displays showed an advantage in terms of pilot performance over the 
basic, or minimum information, displays. Both the standalone and integrated versions of the advanced display 
condition showed significantly faster response times than the basic display conditions for two of the reported 
metrics: total edit time and total response time. In addition, the integrated advanced display supported significantly 
faster initial edit times than the integrated basic display. Surprisingly however, there were no significant findings for 
display location; of the six metrics reported only one, initial response time, showed a near significant difference 
between standalone and integrated. The implication of each metric in the pilot-DAA interaction timeline, along with 
the results for each metric in the present study, is discussed below. 

A. Notification Time 

Notification time is a measure of the time it takes for a pilot to notify ATC of a potential conflict and the need to 
execute a resolution maneuver. A pilot could notify the controller and request a maneuver prior to maneuvering, or 
notify the controller after maneuvering. In this study, pilots took an average of 29 seconds to notify ATC of a 
potential conflict. Interestingly, the average notification time was longer than the average initial response time of 19 
seconds, thus pilots typically began initiating a maneuver prior to notifying ATC. This could be because pilots 
wanted to determine an appropriate maneuver before requesting a clearance, or it could be because pilots were too 
busy determining a maneuver to notify ATC. Overall, pilots notified ATC of a potential conflict and subsequent 
maneuver 97% percent of the time. However, this notification occurred prior to maneuver execution only 65% of the 
time, which indicates that pilots sometimes felt that a potential conflict was too urgent to allow time to request a 
maneuver clearance. 

There were no significant differences in notification time between the four display conditions, although the 
difference between the advanced and basic display approached significance. The slightly faster notification times 
may reflect that the pilots were faster at the entire DAA task (i.e., total response time) when advanced information 
and tools were present in the DAA display. 

B. Initial Response Time 

Initial response time is a measure of the time it takes for a pilot to initiate a maneuver in the GCS control and 
navigation interface after the appearance of an alert on the DAA display. This time includes the time it takes the 
pilot to detect a potential threat. In addition, it may include some time where the pilot is determining an appropriate 
resolution maneuver. Differences in initial response time could be due to differences in the time for pilots to detect a 
potential threat, differences in the amount of time spent determining a resolution maneuver, or differences in control 
and navigation interfaces (see Rorie & Fern 7 for a discussion). Since this study utilized the same control and 
navigation interface for all four display conditions, potential differences would likely be a result of variation in the 
DAA displays. 

There was not a significant difference between the advanced and basic display conditions, which suggests that 
information differences did not affect initial response time. However, there was a near significant difference 
between the standalone and integrated displays, whereby the standalone condition was nearly 5 seconds faster. A 
potential difference here could be explained by pilots initiating a maneuver sooner in the standalone condition in 
anticipation of a potential threat (before it became a self separation or collision avoidance alert on the display) in 
order to compensate for the separate displays. In fact, the distribution of initial response times indicates that pilots 
initiated a maneuver prior to a self separation or collision avoidance alert for 7% of all maneuvers. 

C. Initial and Total Edit Times 

Initial and total edit times reflect the amount of time it takes for a pilot to input and upload a first and final 
maneuver to the aircraft, respectively, from the initiation of that input. In other words, it is the amount of time that 
pilots interact directly with the GCS. Edit times capture most (if not all) of the time spent determining an appropriate 
resolution maneuver; although pilots likely begin to think about a maneuver prior to initiating a response, the lack of 
significant differences in initial response time compared to large significant differences in edit times between the 
basic and advanced information conditions, suggests that pilots execute much of the determine function within the 
edit time. Initial and final upload times are captured because pilots sometimes make multiple uploads to the aircraft, 

15 

American Institute of Aeronautics and Astronautics 



using subsequent uploads to tweak or correct their initial upload. Pilots operating with the advanced displays were, 
on average, 9 seconds faster at inputting and uploading their final resolution maneuvers, although the difference 
between the advanced and basic information display conditions was only significant for the integrated displays at the 
first upload, (5.5s and 17.1s, respectively - a 68% improvement for the advanced display). 

The advanced displays were on average 9 seconds, or 40% faster than the basic displays on total edit time. Thus, 
it appears that the additional information and tools provided by the advanced display conditions assisted pilots in 
more quickly determining and uploading a maneuver in response to displayed self separation and collision 
avoidance alerts. Although the authors hypothesized that the response times overall would also be faster for the 
integrated displays compared to the standalone displays, especially for the advanced integrated display where the 
advanced tools were integrated with the GCS’s control and navigation interface, there were no significant 
differences in either initial or total edit between the standalone and integrated displays. 

Advantages in edit times, do not necessarily equate to overall faster response times, as it only reflects one stage 
of a pilot’s functions to determine and then execute a resolution maneuver. Edit times only capture the time that 
pilots interact directly with the GCS, and not the time prior to interacting with the GCS which may be spent just 
visually examining the DAA display. A slower response earlier in the timeline could negate any benefits found in 
the edit times for a particular display configuration. Total response time captures initial response time in addition to 
total edit time, respectively. 


D. Total Response Time 

Total response time captures the time that it takes for pilots to upload their final maneuver in response to a 
display alert, from when the alert first appears. Total response time captures all three pilot DAA functions and the 
entire pilot-in-the-loop activity of the DAA system. This is the pilot performance metric that DAA system designers 
and standards developers are most concerned with, and the biggest unknown in the complete system. Total response 
time represents the time from the DAA alert appearing until the pilot completes their portion of the DAA timeline 
and the aircraft executes the final, commanded maneuver. Pilots completed their DAA functions about 14 
seconds, or 33% faster with the advanced information displays than with the basic information displays. A 14 
second difference in total response time could have a significant impact on the potential for a well clear violation, 
depending on the rate of closure between the ownship and another aircraft. As mentioned previously, pilots will 
potentially face constrained timeframes for responding. For this experiment, the well clear threshold was set at 40 
seconds to closest point of approach and the alerting threshold at 110 seconds, leaving 70 seconds for the pilot and 
aircraft to respond and maneuver in time to avoid a potential threshold violation. There are essentially two buffers 
on each side of a pilot’s total response time prior to a well clear violation: 1) a buffer for when a potential threat is 
detected (i.e., some allowance for a later detection due to sensor performance or encounter geometry), and 2) a 
buffer for aircraft performance (and lower restrictions on aircraft performance reduce costs on manufacturers and 
allow a wider variety of aircraft to meet defined standards and regulations). Shorter total response times for a pilot to 
respond to a DAA alert could allot for greater buffers for late detections and more limited aircraft performance. 

E. Summary 

Overall, the metrics extracted from the proposed eight stages of pilot-DAA interaction appear to be useful in 
quantifying and understanding the pilot DAA task functions of detecting potential threats, determining a resolution 
maneuver, and executing that maneuver via the GCS command and control interface. In this study, these metrics 
were sensitive primarily to differences in the amount and kind of information between the advanced and basic 
display conditions. The advanced displays resulted in faster response times than the basic displays for three of the 
five reported metrics. Most importantly, the advanced information displays showed significantly faster total 
response times than the basic information displays, a metric that captures the entire pilot DAA task, from the 
presentation of a self separation or collision avoidance alert to execution of the final resolution maneuver. This 
finding could have important implications for the development of minimum information requirements for DAA 
displays. The basic information condition in this study represented a first attempt to establish a minimum 
information set, however, it’s poorer performance in terms of slower pilot response times compared to the advanced 
information displays could indicate that the information presented to pilots was not sufficient. As noted earlier, 
delayed pilot response times increase the potential of violating separation and collision avoidance thresholds. 

It is important to note here that the results of this study alone do not yet support any final conclusions. First, the 
results of this study need to be assessed in conjunction with the loss of separation or well clear results reported by 
Santiago & Mueller 4 . Although faster pilot response times are considered more desirable, this is only true if the 
faster response times coincide with fewer losses of well clear; if for some reason pilots are faster but cause more 

16 

American Institute of Aeronautics and Astronautics 



losses of well clear, the information provided in the display needs to be reassessed. In addition to examing the loss 
of well clear results, a safety analysis is required to determine whether the observed level of pilot performance (both 
in terms of response times and well clear violation rates) meet a defined airspace safety threshold. Second, although 
the results suggest that the minimum information set from this study may not be sufficient, it does not automatically 
lead to the conclusion that the full advanced information suite is a necessary requirement for DAA displays. It is 
possible that only one or two features of the advanced displays contribute to improved pilot performance and are 
required in the minimum information set. Finally, the lack of significant differences between the standalone and 
integrated condition, despite the hypothesis that the integrated displays would support faster response times overall, 
also begs additional investigation. Before accepting the null hypothesis, that there was no performance impact 
between the two display configurations, a number of additional operational conditions need to be examined. For 
example, when the DAA display and the GCS tactical situation display or moving map are at different zoom levels 
or different orientations. Or when the DAA display is in different locations within the GCS. Clearly, further work is 
needed to identify what constitutes the minimum display requirements that will satisfy the airspace safety threshold. 


VI. CONCLUSION 

The simulation experiment described above presents a productive first step in the vast amount of work needed by 
the UAS community to quantify the minimum operational performance standards for a UAS DAA system. 
Specifically, this study quantified the pilot response time on the DAA task in order to contribute to defining the 
minimum information that a pilot needs presented on a DAA display to maintain self separation and collision 
avoidance from other aircraft. The first of several planned studies, the development of the simulation capabilities 
and initial display concepts, including the algorithms underlying the advanced information tools, facilitate a number 
of follow on studies and fruitful research opportunities to support continued investigation of issues critical to the 
integration of UAS into the NAS. Future studies will need to further the work presented here by examining the 
effect on pilot performance of some of the individual advanced information features tested in this study, different 
surveillance sensor parameters and performance, alternative ways of display resolution information and guidance, 
alternative DAA algorithms, alternative GCS configurations (including different control and navigation interfaces), 
communication delays, automation, and more. 
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